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(54) Content-based video story browsing system 

(57) A content-based video story browsing system 
is disclosed including setting and representing relations ..>- 
between objects and places of the story. A content- 
based video story browsing system includes the steps 
of selecting objects and places as factors organizing 
contents of a video story and setting a relational graph 
relating the objects to corresponding places; setting a 
video story section and a representative scene as an 
events of contents development corresponding to the 
relations of the objects and the places: and displaying 
the video story section and the representative scene 
and its contents as browsing information corresponding 
to the relation selected from the relational graph. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to a video 
story browsing system and more particularly to a con- 
tents-based video story browsing system defining and 
representing the relationship of significant factors in a 
video story. The factors includes an object, a place, 
and/or an event in time. The present invention- also 
relates to a video story browser representing an overall 
outline consisting of paragraphs summarizing signifi- 
cant events for a convenient video story browsing. 

Discussion of Related Art 

[0002] Video browsers and video story summariza- 
tion in the related art generally utilize key frames to rep- 
resent a large amount of story contents. Thus, it is 
difficult and/or complicated to summarize and browse 
the overall content of a movie or drama by extracted key 
frames or a character-based story browsing. Moreover, 
presenting contents of some scenes is ambiguous, 
making it difficult browse the exact information. 
[0003] Although picture images of a drama or a 
movie progress by interconnection of events, key frame 
browsing or character-based browsing is simply per- 
formed without presenting any interconnected relations 
of events. A video story browsing technique has been 
proposed for summarizing and browsing the overall 
contents by extracting key frames from dynamic images 
and graphically showing the time relationship of each 
key frames. This technique may be usable for a simple 
video file, but is not suitable for video files of huge vol- 
ume such as a drama or movie with numerous intercon- 
nected events over time. Moreover, because picture 
images of greater volume produce a large number of 
key frames, it is difficult to graphically represent key 
frames. Accordingly, it is hard for a user to understand 
the graphically represented information. 
[0004] Another technique of key frame-based video 
story summarization and browsing has been proposed 
in which classes are formed according to a length of a 
section from which a key frame is extracted. Generally, 
a small number of key frames are extracted from sec- 
tions of longer lengths for higher classes and a greater 
number of key frames are extracted from sections of 
smaller sections for lower classes, thereby forming a 
tree structure. However, this structure cannot be used 
comprehensively to browse the overall contents with 
representative scenes because the key frames are 
extracted from a too large section for the higher classes, 
making it difficult to retrieve a desired part using the key 
frames. 

[0005] Still another object-based browsing tech-' 
nique has been proposed in which objects of dynamic 



images are organized and an attribute of each object in 
a section is displayed. However, this technique also 
makes it difficult to understand the overall contents. In 
addition, it is often difficult to determine which attribute 
s should be designated for each object during a search 
algorithm. 

[0006] Therefore, since video stories such as a 
drama or movie progress by changing characters and 
places in time, it would not only be impossible to sum- 
10 marize the contents, but would be difficult to browse the 
overall convents just by watching extracted characters 
or scenes. 

SUMMARY OF THE INVENTION 

15 

[0007] Accordingly, an object of the present inven- 
tion is to solve at least the problems and disadvantages 
of the related art. 

[0008] An object of the present invention is to pro- 

20 vide a content-based video story browsing system. 
[0009] Another object of the present invention is to 
provide a video story browsing system summarizing the 
contents of a video story according to the relations of an 
object, a place, an event, and time. 

25 [001 0] Still another object of the present invention is 
to provide a content-based video story browsing system 
allowing a browsing by a relational graph reflecting a 
development of events in time with respect to a<main 
object and place in a movie or drama. 

30 -^001 1 ] A further object of the present invention is to 
provide a content-based video story browsing system 
allowing a browsing by sections of a movie or drama, 
each section describing events with respect to an object 
and a place in the section of the movie or drama. 

35 [001 2] Still a further object of the present invention 
is to provide a content-based video story browsing sys- 
tem allowing a browsing by relation of a main object and 
a place in which one relation is represented by a frame 
and description corresponding to the section of the 

40 story containing the relation. 

[0013] Still a further object of the present invention 
is to provide a contents-based video story browsing sys- 
tem allowing browsing a video story based upon con- 
tents of a video story expressed by relations of a main 

45 object, a place, and an event, in which relations of plural 
objects and a place as well as the relations of one object 
and one place can be browsed. 
[0014] Still a further object of the present invention 
is to provide a contents -based video story browsing sys- 

so tern allowing summarization and browsing of a video 
story by displaying the contents of the video story in 
event paragraphs, in the form of a text or a text and rep- 
resentative scenes. 

[001 5] Additional advantages, objects, and features 
55 of the invention will be set forth in part in the description 
which follows and in part will become apparent to those 
having ordinary skill in the art upon examination of the 
following or may be learned from practice of the inven- 
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tion. The objects and advantages of the invention may 
be realized and attained as particularly pointed out in 
the appended claims. 

[0016] To achieve the objects and in accordance 
with the purposes of the invention, as embodied and 
broadly described herein, the present invention dis- 
closes a contents-based video story browsing system 
based upon relations of objects and places. The con- 
tents-based video story browsing system includes a 
scheme indicating a video story section and a repre- 
sentative scene as information for the development of 
the story events. 

[0017] According to another embodiment of the 
present invention, a contents-based video story brows- 
ing system includes steps of selecting objects and 
places as factors organizing the contents of a video 
story and setting a relational graph relating the objects 
to corresponding places; setting a video story section 
and a representative scene as an event of contents 
development corresponding to the relations of the 
objects and the places; and displaying the video story 
section and the representative scene and its contents 
as the browsing information corresponding to the rela- 
tion selected from the relational graph. 
[0018] According to another embodiment of the 
present invention, a contents -based video story brows- 
ing system includes the steps of setting a relational 
graph of main objects and places as factors organizing 
contents of a video story, and adding an event descrip- 
tion of text along with position information of sections 
corresponding to the relation and a representative 
frame position information representing each section; 
displaying an object-place relational graph of the video 
story; and it at least one relation is selected from the 
relational graph, displaying a representative frame of 
each section in the selected relation and an event 
description based upon the position information, and 
screening a user-desired section. 
[0019] The contents-based video story browsing 
system further includes a step of setting the object- 
place relational graphs with respect to the overall video 
files and each video file, and schematically browsing the 
overall contents of several video files from the relational 
graphs with respect to the overall video files and brows- 
ing in detail each video file. 

[0020] According to another embodiment of the 
present invention, the overall outline of a video story is 
divided by significant events and event paragraphs rep- 
resenting each events, and a scheme indicating a video 
story section corresponding to each event paragraph is 
set. If a certain event is selected, the information about 
the corresponding story section is displayed in 
response to the selection. 

BRIEF DESCRIPTION OF THE ATTACHED DRAW- 
INGS 

[0021 ] The invention will be described in detail with 



reference to the following drawings in which like refer- 
ence numerals refer to like elements wherein: 

FIG. 1 is a flow chart showing the steps of forming 
5 the browsing information for browsing a story 
according to the present invention; 
FIG. 2 is an example of a graphic data structure set- 
ting the relation of an object and a place according 
to the present invention; 
10 FIG. 3 is a block diagram for a video story produc- 
tion and a video story browser according to present 
invention; 

FIG. 4 is an example of a user interface with a 
graphic data structure setting the relation of an 
15 object and a place according to the present inven- 
tion; 

FIG. 5 is another example of a user interface with a 
graphic data structure setting the relation of an 
object and a place according to the present inven- 

20 tion; 

FIG. 6 is an example of a user interface when sev- 
eral objects and places are selected in a graphic 
data structure setting the relation of an object and a 
place according to the present invention; 

25 FIG. 7 is an example of a user interface represent- 
ing an events in the overall story as a paragraph 
and representing key frames and their time rela- 
tions according to the present invention; 
FIG. 8 is an example of a user interface represent- 

30 ing a events in the overall story as a paragraph and 
presenting a scene representing the events as a 
key frame according to the present invention; 
FIG. 9 is an example of a user interface represent- 
ing a events in the overall story as a paragraph and 

35 presenting the relationship of characters and 
places related to the events, and presenting repre- 
sentative scenes according to the present inven- 
tion; 

FIG. 10 is an example of a user interface showing 
40 representative scenes by respective event para- 
graphs form the story according to the present 
invention; 

FIG. 1 1 shows a structure of browsing information 
for embodying of FIGS. 9 and 10; 

45 FIGS. 12 and 13 are each examples of a user inter- 
face showing events in the overall story as a para- 
graph and presenting the multiple level of 
characters and places related to the events; and 
FIGS. 14 to 16D illustrate example data structures 

so for a content based browsing system according to 
the present invention. 

DETAILED DESCRIPTION OF PREFERRED EMBOD- 
IMENT 

55 

[0022] Reference will now be made in detail to the 
preferred embodiments of the present invention, exam- 
ples of which are illustrated in the accompanying draw- 
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ings. 

[0023] FIG. 1 is a flow chart for developing the data- 
base used in a video story browser. Generally, such a 
database is formed by setting a plurality of relations 
between objects and a places in a video story such as a 
movie or a drama. As shown in FIG. 1, characters and 
places necessary to develop events in a story are set as 
main objects and main places, respectively (step 101). 
[0024] Subordinate classes to main objects and 
main places may also be defined. For example, if the 
main place is set to a "hospital," scenes may be subdi- 
vided as subordinate classes into a corridor, a ward, a 
surgery room, a lobby, and a hospital garden. However, 
if representing the main place simply as a "hospital" 
would better represent the overall outline of the video 
story, the subordinate classes are combined to form 
simply one main place, namely the "hospital." 
[0025] After setting the main objects and main 
places, relationships of the main objects to the main 
places are set (step 102). For example, if character A, 
character B, and character C are set as the main 
objects; and if a hospital, an indoor, and an outdoor are 
set as the main place, the relations of the objects and 
places may be set as follows: places where character A 
appears are the hospital and the indoor; places where 
character B appears are the indoor and the outdoor; 
and a place where character C appears is the outdoor. 
[0026] The set relationships reflect the sections of 
the video story in which objects actually appear in the 
video story. Accordingly, the sections in which objects 
appear with respect to places are defined and indexed 
using the set relationships. A section may signify a 
frame in which an object actually appears or may signify 
an interval of frames in which an object generally 
appears. For example, characters A and B may alter- 
nately appear in a section where characters A and B are 
carrying on a conversation. Thus, neither characters A 
or B actually appears for the full duration of the frame 
interval. However, the frame interval would be indexed 
as a section in which either character A or B appears. 
[0027] Thereafter, a representative scene for each 
indexed section are set (step 103). A frame of each sec- 
tion defined according to the relationships set in step 
102 which best represents the corresponding section is 
set as the representative scene. The frame location of 
the representative scene is marked. In the example 
above," a frame showing an operation may be set as the 
representative scene of a section in which character A 
comes into the hospital with an illness, and the location 
of that frame would be marked. 
[0028] Finally, the events are illustrated for each 
section by adding appropriate notes to corresponding 
sections (step 104). According to the above example, 
the operation frame is set as a representative scene for 
the relation of character A and the hospital, and notes 
which summarize the story contents of the section are 
added as text. 

[0029] As described above, the relations of the 
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main objects and main places are established based 
upon the contents of the video story. Moreover, the 
basic information of object, place, event, and time for a 
video story browsing is complete by indexing relevant 

5 sections, setting representative scenes in each section, 
and adding notes to each section. In the preferred 
embodiment, the developed information is stored in the 
header of a multimedia stream. However, the informa- 
tion may be stored in other locations. As a result, the 

70 video story browsing system allows users to perform a 
content-based browsing of a video story by reading the 
information stored in the header of the multimedia 
stream. 

[0030] FIG. 2 shows a graphical data structure of 

is the relations between objects and a place/event devel- 
oped by the steps described in reference to FIG. 1. 
Referring to FIG. 2, character A 201, character B 202, 
and character C 203 are set as the main objects; and a 
hospital 204, an indoor 205. and an outdoor 206 are set 

20 as the main places. Character A has relations to the 
hospital 204 and the outdoor 206, character B has rela- 
tions to the hospital 204 and the indoor 205, and char- 
acter C has a relation to the indoor 205. The relations of 
the objects 201 to 203 and places 204 to 206 are graph- 

25 ically shown by links such as the link 207. 

[0031] The sections in which each character 
- appears are indexed according to the object-place rela- 
tions. As shown in FIG. 2, three sections 1 to 3 (208, 
209 and 210) in which character A appears in the hospi- 

30 -ial 204 would correspond to the relation between char- 
acter A and the hospital 204. A scene representing each 
of the three sections, the start and the end positions of 
the section, and the event illustration of each section are 
also illustrated. 

35 [0032] Particularly, in section 2 (209) character A 
appears from frame 16 to frame 21. Accordingly, frame 
18 becomes a key frame 211 as the representative 
scene and its position information Frm_K is recorded. 
The frame 1 6 is a start frame 2 1 2 of section 2 (209) and 

40 its position information Frm_S is recorded, and the 
frame 21 becomes an end frame 213 of section 2 (209) 
and its position information Frm_E is recorded. A sum- 
mary of the story contents or event for section 2 (209) is 
stored as text Event(text) 214. 

45 [0033] A human object scheme 215 which is a. 
lower class subordinate to a main object may be defined 
for the main objects 201 to 203, and a place object 
scheme 216 which is a the lower class subordinate to a 
main place may be defined for the main places 204 to 

so 206. Namely, the place object scheme 216 may define a 
ward, a corridor, a lobby, a surgery room, etc. as the 
lower class places of the hospital 204 and object-place 
relations would be set according to the defined lower 
class places rather than the hospital 204. 

55 [0034] The graphic structure shown in FIG. 2 is only 
a portion of the object-place/event relations, in an actual 
structure, there may be any number of sections with 
respect to the relation of one object and one place; 
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there may be any number of objects having relations to 
one place; and there may be any number of places hav- 
ing relations to one object. To represent a scene without 
any objects, the structure may represent a place having 
a relation with a null object. There also may be a multi- $ 
pie object-place relation as well as the single object- 
place relation. 

[0035] FIG. 3 shows a first embodiment of a video 
story browsing system implementing the database dis- 
cussed in reference to FIGs. 1 and 2. To facilitate the 
explanation of the system, the system for the video sup- 
plier and the system for the user have been separately 
illustrated. The video supplier develops a browsing 
schemes such as shown in FIG. 2 by the steps in FIG. 
1, and the user may browse a video story based upon 
the browsing schemes. 

[0036] Particularly, digital video data 301 is tempo- 
rarily stored in a video buffer 302, and a video process- 
ing unit 303 controls the digital video data 301 stored in 
the video buffer 302 as a multimedia video stream file. A 
control processing unit 304 produces or reads the nec- 
essary information from the digital video data 301. A 
user interface and display unit 305 displays the video 
data processed by the video processing unit 303 and 
the control processing unit 304, and provides an inter- 
face to receive an input from a user for production of a 
browsing scheme. 

[0037] A browsing scheme such as the object- 
place/event relation schemes of FIG. 2 is processed by 
the control processing unit 304 and temporarily stored 30 
in a memory for video information 206. Theprocessed 
schemes are then stored in a storage 307 in a proper 
format along with the video data stored in the video 
buffer 302. The video data stored in the storage 307 and 
the browsing schemes are provided to the user through 35 
a recording media, a wire-broadcasting, or a radio- 
broadcasting network. 

[0038] A user browses and. views the video data 
and object-piace/event data stored in the storage 307. A 
video buffer 308 temporarily reads and stores the video *o 
data from the storage 307. A video processing unit 309 
and a control processing unit 310 processes the video 
data stored in the video buffer 308. Particularly, a mem- 
ory for video information 312 reads and temporarily 
stores the object-place/event data from the storage 307. 45 
The control processing unit 310 processes the informa- 
tion from the memory for video information. Subse- 
quently, the processed video data and information are 
displayed according the browsing selections received 
through a user interface and display unit 311. so 
[0039] Therefore, in the content-based video story 
browsing system of FIG. 3, the procedure illustrated in 
FIG. 1 and the resulting data structure of FIG. 2 are 
completely organized through a user interface and dis- 
play unit 305 by the supplier. The database is stored in 55 
an appropriate storage 307 along with the video data. A 
user may then perform video story browsing through a 
user interface and display unit 31 1 
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[0040] FIG. 4 shows an example of a user interface 
for the user of the video story browser according to the 
present invention. The user interface corresponds to the 
relations of object-place/event of FIG. 2. The. video story 
browser 400 displays an object-place relational graph 
401 and relation contents 402 illustrating the story con- 
tents of sections corresponding to a selected relation. 
The object-place relations developed are as follows: 
character A 403 has a relation to a hospital 406 and an 
outdoor 40.8; character B 404 has a relation to the hos- 
pital 406 and an indoor 407; and character C 405 has a 
relation to the indoor 407. 

[0041] While browsing, if a user selects the relation 
between character B and the hospital 406 by selecting a 
link 409 or by directly selecting the character B and the 
hospital 406 through the user interface and display unit 
311, the control processing unit 310 processes to dis- 
play the four sections 1 to 4 (410) and events 41 1 corre- 
sponding to the above relation, by referring to the 
memory 312. The four sections 1 to 4 (410) are the rep- 
resentative scenes selected from corresponding sec- 
tions. Also, the notation " over the link 409 indicates 
the four existing sections in the relation between char- 
acter B and the hospital 406. However, the number of 
sections need not be indicated, but if indicated, any 
other method may be used. 

[0042] The user may select objects or a place 
rather than a relational link 409. If at least one object is 
selected from the relational graph, sections with the 
object(s) can be browsed and displayed regardless of 
the places. Likewise, if a certain place is selected, sec- 
tions with that place can be browsed regardless of the 
objects. 

[0043] FIG. 5 is an example of the user interface 
when one of the section from the sections 1 to 4 (410) 
corresponding to the object-place relation (409) in FIG. 
4 is selected by selecting the representative scene cor- 
responding to section 2. Thus, if a user selects the rela- 
tionship between character B and the hospital 406, and 
then selects section 2, a more detailed description 51 1 
of section 2 and the representative scene are displayed 
to the user in the relation contents 502. 
[0044] The video story browser still displays an 
object-place relational graph 501 with the data structure 
of the characters 503 to 505, the places 506 to 508, and 
the selected link 509. The user may reproduce a portion 
ol the video data corresponding to section 2 through an 
icon 512. In response, the start frame to the end frame 
of the corresponding section are displayed. 
[0045] FIG. 6 is an example of a user interface 
when more than one object and a place are selected 
from the relational graph. The video story browser 600 
displays an object-place relational graph 601 and rela- 
tion contents 602. The relational graph 601 includes a 
data structure of characters 603 to 605, the places 606 
to 608, and links 609 and 610. If characters A and B and 
a place 606 are selected, representative scenes 61 1 
and 613 with illustrations 612 and 614 are displayed. 
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The representative scenes 611 and 613 correspond 
with sections in which character A and character B 
simultaneously appears in the hospital 606. 
[0046] As discussed above, if plural objects are 
selected without selecting a place, the video story 
browser 600 displays sections in which the selected 
objects appear regardless of the places. Thus, the user 
may view the places in which each selected object 
appears. Similarly, if one place is selected without 
selecting an object, the video story browser 600 dis- 
plays the sections in which the selected place appears 
regardless of objects. 

[0047] FIG. 7 shows an example of a user interface 
representing the events of a story by paragraphs and 
representing the relations of the events, including rela- 
tions in time, by graph in key frames. The video story 
browser 700 has a story display 701 with event para : 
graphs 701a and when one event paragraph is selected, 
the section contents screen 702 shows a transition 
graph linking scenes 1 to 4 of the section corresponding 
to the selected event. 

[0048] FIG. 8 is an example of a user interlace rep- 
resenting events of the story by paragraphs and pre- 
senting scenes representing the events in the key 
frames. The video story browser 800 has a story display 
801 with event paragraphs 801a and when one event 
paragraph is selected, the section contents screen 802 
shows scenes 1 to 7 of the section corresponding to the 
selected event with brief description for each scene. 
[0049] FIG. 9 is an example of a user interface rep- 
resenting events of the story by paragraphs and pre- 
senting the relations between the characters and places 
as well as representative scenes of the corresponding 
event of the story. The video story browser 900 has a 
story display 901 representing event paragraphs 901a 
and when on event paragraph is selected, the section 
contents screen 902 shows representative scenes for a 
character-place relational graph of a section corre- 
sponding to the selected event. 
[0050] A character-place relational graph screen 
903 and a representative scene screen 904 for a certain 
relation are displayed in section contents screen 902. 
When one event paragraph 901a corresponding to a 
certain event is selected in the story display 901, the 
relations of characters A, B and C (905 to 907) appear- 
ing in section corresponding to the selected event and 
places such as a hospital 908, an indoor 909, and out- 
door 910, are.displayed in the character-place relational 
graph screen 903. If one relation 91 1 between character 
B and the hospital 908 is selected, representative 
scenes oi a section in which character B appears in the 
hospital 908 are displayed in the representative scene 
screen 904. 

[0051] FIG. 10 is an example of a user interface 
showing. representative scenes for the respective event 
paragraphs of the story. The video story browser 1000 
has a story display 1001 representing key frames 14 (or 
representative scenes) and paragra phs c orresponding 



to the events and when one event paragraph is 
selected, the section contents screen 902 show repre- 
sentative scenes for a character-place relational graph 
of a section corresponding to the selected event. A 
5 character-place relational graph screen 1 003 and a rep- 
resentative scene screen 1004 of a certain relation are 
displayed in the section contents screen 1002. 
[0052] The browsing method and results displayed 

• in the section contents screen 1002 are the same as 
10 described in reference to FIG. 9. However, in FIG. 10, a 

representative key frame 14 representing an event may 
be selected rather than an event paragraph to display 
the character-place relation and the representative 
scenes corresponding to the selected key frame 14. 
15 [0053] FIG. 1 1 is another example of a user inter- 
face representing a character-place relational graph 
according to. the present invention. Reference numeral 
1101 denotes the overall story consisting of event para- 

• graphs 1101a. Reference numeral 1102 denotes 
20 scenes representing an event section by key frames (or 

position information) 1 102a and links 1104 to the event 
paragraphs of the story section. 
[0054] Reference numeral 1103 denotes a graphic 
structure of the relations between characters (1103a, 

25 1 103b) and place (1 103c, 1 103d) in each section corre- 
sponding to the event paragraphs 1101a. Character 
schemes 1103b subordinate to character 1103a are 
defined as the lower class relational schemes to the 
characters 1103a. and place schemes 1103d subordi- 

30 >nate to places 1103c are defined as the lower class 
relational schemes to the places 1103c. For example, a 
"ward 1 , a 'surgery room', and a 'hospital corridor' may be 
subordinate to the 'hospital'. 
[0055] FIGS. 12 and 13 are each examples of a 

35 user interface showing events of a story by paragraphs 
and presenting the multiple level of characters and 
places related to the events. As shown in FIG. 12, the 
video story browser 1200 has an object-place relational 
display 1201 and a story display 1202 representing 

40 event paragraphs. The object-place relational display 
1201 includes a screen 1203 graphically representing 
the relations of objects and places, a representative key 
frame 1204 corresponding to a selected relation, and a 
description 1205 of the event in the section. 

45 [0056] A representative scene (key frame) repre- 
senting the event, as in FIG. 10, may be displayed in the 
story display 1202. In FIG. 12, three characters A, B and 
C are set as the objects, and a hospital, a character A's 
house, a park are set as the places. Particularly, charac- 

so ter A has a relation to the 'hospital' and the 'park', and 
character B has a relation to the 'hospital', and charac- 
ter C has a relation to character A's home. 
[0057] When a relation 1206 between character B 
and the 'hospital' is selected, a representative scene of 

55 a section in which character B appears in the 'hospital' 
is displayed as a key frame 1204 with a description 
1205 about the section. A display-zoom button 1207 
indicates that there is a lower class level subordinate to 
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the selected place, namely the 'hospital*. The display- 
zoom button 1207 may be provided in a menu, rather 
than being displayed in the section contents screen 
1 201 . If a user selects the display-zoom button 1 207, an 
object-place relation of a level subordinate to 'hospital' 5 
would be displayed, as shown in FIG. 13. 
[0058] Referring to FIG. 1 3, the video story browser 
1300 has a section contents screen 1301 and a story 
display 1302. The section contents screen 1301 
includes an object-place relational graph 1303, a key 10 
frame 1304 corresponding to the selected relation, and 
a description 1305 of the event in the section. The lower 
class places defined for the 'hospital' are the lobby of 
the hospital and a ward of the hospital. Character B has 
relations 1306a and 1306b to both the lower class 15 
places. 

[0059] Therefore, a user may select the relation 
1306a between character B and the lobby or relation 
1306b between character B and the ward from a video 
story browser 1 300 of FIG. 1 3. In response, a key frame 20 
1304 and its description 1305 corresponding to the 
selected relation would be displayed. If the button 1307 
grouping the lobby and the ward is selected, the screen 
returns to the upper level relational graph as shown in 
FIG. 12. 25 
[0060] As described above in reference to FIGS. 1 
to 13, the present invention is capable of browsing a 
video story by the relations of object-place. FIGS. 1 to 
13 concern a single video file, a single drama or a single 
movie. -so 
[0061] The present invention employs the concept 
of the content-based video story browsing system. The 
plural video files are represented via a single relational 
graph, and the contents of each video file are repre- 
sented via a single relational graph. The contents can 35 
be briefly browsed from the relational graph, and a 
detail browsing may be performed from the relational 
graph for the contents of one video file. If a movie is 
divided into several video files, different events develop 
with the same objects (e.g. the same hero or heroin) 40 
and same places (the main places of the development 
of the video story). The divided video files are consid- 
ered as a single video file, and its contents are illus- 
trated by the object-place relational graph as shown in 
FIGS. 1 to 13. Thus, a user may schematically browse 45 
the overall contents of the several video files of a single 
movie. 

[0062] FIG. 1 4 shows an example of the data struc- 
ture for a story browser according to the present inven- 
tion. As shown, a video is divided relation information so 
which includes object and place relation-information and 
scene section informations. Each scene section infor- 
mation includes time period and contents of the section 
in the story. FIG. 15 shows another example of the data 
structure for a story browser according to the present 55 
invention. The video is divided into scene section infor- 
mations rather than the relation information. Each 
scene section information includes a time period infor- 
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mation, contents information, and place information as 
well a plurality of object information. 
[0063] FIG. 16A to 16D shows the data structures 
for the time period, object, place and contents informa- 
tion in FIGS. 14 and 15. Particularly, the time period 
information includes the start frame and the end frame 
of a section, as shown in FIG. 16A. The object informa- 
tion includes name and a representative frame, as 
shown in FIG. 16B. The place information includes the 
name and representative frame, and may optionally 
include lower class places, as shown in FIG. 16C. The 
contents information includes the text story and the rep- 
resentative frame, as shown in FIG. 16D. 
[0064] According to the present invention, the over- 
all contents of a video story are summarized and a cer- 
tain position in the summary may be selected to browse 
the detailed contents corresponding to that position. In 
addition, since the present invention develops the rela- 
tional graph based upon the basic factors, namely the 
characters and backgrounds, necessary for the devel- 
opment of an event, a user can easily understand the 
contents. 

[0065] Moreover, this invention may be applicable 
to a video story of animals set as the main objects 
rather than persons. Also, the level of places may be 
varied with the development of the story. Thus, the 
present invention allows to easily summarize the con- 
tents of the story with development of an event since a 
lower class of an object or place subordinate to the main 
object and place may be defined. 
[0066] In addition, the present invention sets the 
relationship of objects, places, and time from the 
according to the development of an event, and various 
modifications are available. For example, for a video 
story of sports, e.g. golf, the main object may be set as 
a golfer, and the main place may set as a group of a tee- 
ing ground, a fairway, etc. Since the event of each scene 
may be. related to the kinds of the golfer's swings, the 
present invention may be effectively implemented in 
sports broadcasting. 

[0067] Furthermore, the present invention includes 
a data structure in which the contents of several video 
files are represented via a single relational graph, and 
another data structure in which the contents of each 
video file are represented via a single relational graph, 
such that a user may schematically browse the overall 
contents of the several video files from the relational 
graph and browse in detail the contents of a single video 
file from the corresponding relational graph, thus assur- 
ing a package browsing of the several video files. 
[0068] According to the present invention, the over- 
ail story of a drama or movie is illustrated by a collection 
events in the drama or movie, thus removing the short- 
comings of the conventional key frame-based contents 
summarization and browsing method which describes 
the development of a video story only with representa- 
tive scenes. Because the overall story of a drama or 
movie is represented by event paragraphs, the present 
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invention removes the complexity and the difficulty of 
expressing the overall contents of large scale solely by 
key frames or other conventional browsing method, thus 
ensuring easy and simple summarization and browsing 
of a video story. 

[0069] The foregoing embodiments are merely 
exemplary and are not to be construed as limiting the 
present invention. The present teachings can be readily 
applied to other types of apparatuses. The description 
of the present invention is intended to be illustrative, and 
not to limit the scope of the claims. Many alternatives, 
modifications, and variations will be apparent to those 
skilled in the art 

Claims 

1 . A content-based video story browsing system com- 
prising: 

a memory storing a video story and browsing 
data based upon relations between at least one 
object and at least one place of the video story; 
a user interface displaying the browsing data, 
receiving user selections based upon the dis- 
played browsing data, and displaying a section 
of the video story corresponding to the user 
selections. 

2. A system of claim 1, wherein the displayed brows- 
ing data comprises. 

a relational graph screen showing the relations 
between at least one object and at least one 
place of the video story; and 
a relation contents screen showing at least one 
representative scene and brief description of a 
section in the video story corresponding to at 
least one selected relation from the relational 
graph screen. 

3. A system of claim 1. wherein the displayed brows- 
ing data comprises: 

a story screen showing at least one event of 

the video story by paragraphs; and 

a section contents screen showing one of 

a transition graph linking representative 
scenes of a section in the video story cor- 
responding to a selected event from the 
. story screen, 
representative scenes and brief descrip- 
tion of a section in the video story corre- 
sponding to a selected event from the story 
screen, 

a relational graph of the relations between 
at least one object and at least one place 
of a section in the video story correspond- 



ing to a selected event from the story 
screen, and at least one representative 
scene of a section corresponding to a 
selected relation from the relational graph, 

5 or 

a relational graph of the relations between 
at least one object and at least one place 
of a section in the video story correspond- 
ing to a selected event from the story 

w screen, and at least one representative 

• scene and brief description of a section 
corresponding to a selected relation from 
the relational graph. 

is 4. A system of claim 1 , wherein the displayed brows- 
ing data comprises: 

a story screen showing at least one event of 
the video story by paragraphs and a represent- 
20 ative scene corresponding to each one of said 

at least one event; and 

a section contents screen showing a relational 
graph of the relations between at least one 
object and at least one place of a section in the 
25 video story corresponding to a selected event 

from the story screen, and showing at least one 
representative scene of a section correspond- 
ing to a selected relation from the relational 
graph. 

30 .>* 

5. A method for developing the data structure for a 
content-based video story browsing system com- 
prising the steps of: 

35 selecting at least one object and at least one 

place as factors organizing the contents of a 
video story and setting the selected objects 
and places as the main objects and main 
places; 

40 setting relations between the main objects and 

main places; 

setting at least one representative scene of a 
section in the video story corresponding to 
each set relations; 
45 setting a description to each of said section in 

the video story; and 

scoring the set information in a storage to be 
used in a content-based video story browsing. 

so 6. A method of claim 5, further comprises the steps of: 

setting low class levels subordinate to at least 
one of either the set main objects or the set 
main places; and 
55 setting relations between said low class levels 

subordinate to at least the set main objects and 
the set main places or setting relations 
between said low class levels subordinate to at 
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least set main places and the set main objects, 
respectively. 

7. A content-based video story browsing method com- 
prising the steps of: 

(a) displaying a browsing data based upon 
relations between at least one object and at 
least one place of the video story; 

(b) receiving user selections based upon the 
displayed browsing data, and displaying a sec- 
tion of the video story corresponding to the 
user selections. 

8. A. method of claim 7, wherein step (a) comprises 
the step of displaying a relational graph showing the 
relations between at least one object and at least 
one place of the video story; and step (b) comprises 
displaying at least one representative scene and 
brief description of a section in the video story cor- 
responding to a selected relation or relations from 
the relational graph. 

9. A, method of claim 8, wherein step (a) comprises 
the step of displaying a story screen showing at 
least one event of the video story by paragraphs; 
and wherein step (b) comprises displaying a transi- 
tion graph linking representative scenes of a sec- 
tion in the video story corresponding to a selected 
event from the story screen. 

10. A display of a content-based video story browsing 
system comprising: 

a story screen showing at least one event of 
the video story by paragraphs; and 
a section contents screen showing a section of 
the video story corresponding to a selected 
event from the story screen. 

11. A display of claim 1 0, wherein the section contents . 
screen shows a transition graph linking representa- 
tive scenes of the section in the video story corre- 
sponding to a selected event from the story screen. 

12. A display of claim 10, wherein section contents 
screen shows a relational graph of the relations 
between at least one object and at least one place 
of a section in the video story corresponding to a 
selected event from the story screen, and show at 
least one representative scene of a section corre- 
sponding to a selected relation from the relational 
graph: 

13. A display of claim 10, wherein the story screen 
shows a representative scene corresponding to 
each one of said at least one event: and wherein 
the section contents screen shows a relational 



graph of the relations between at least one object 
and at least one place of a section in the video story 
corresponding to a selected event from the story 
screen, and shows at least one representative 
5 scene of a section corresponding to a selected rela- 
tion from the relational graph. 
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